Balancing Precision and Retention
in Experimental Design

 

Gustavo Diaz
Northwestern University

gustavodiaz.org

Erin Rossiter
University of Notre Dame

erossiter.com

 

Paper and slides: gustavodiaz.org/talk

Bias-variance tradeoff as darts

But the game of darts is more complicated

Two types of tradeoffs

  1. Explicit: Is some bias worth the increase in precision?

  2. Implicit: Improving precision without sacrificing unbiasedness?

Two types of tradeoffs

  1. Explicit: Is some bias worth the increase in precision

  2. Implicit: Improving precision without sacrificing unbiasedness?

 

Cost has to come from somewhere else!

Improving precision in experiments

Standard error of estimated ATE in conventional experimental design (Gerber and Green 2012, p. 57)

\[ SE(\widehat{ATE}) = \sqrt{\frac{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}{N-1}} \]

Improving precision in experiments

\[ SE(\widehat{ATE}) = \sqrt{\frac{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}{N-1}} \]

Improving precision in experiments

\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{N-1}} \]

Variance component

Decrease \(SE(\widehat{ATE})\) with alternative research designs

Improving precision in experiments

\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{N-1}} \]

Variance component

Decrease \(SE(\widehat{ATE})\) with alternative research designs

Block-randomization

Repeated measures

Pre-treatment covariates

Pair-matched design

Online balancing

Sequential blocking

Rerandomization

Matching

Improving precision in experiments

\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{N-1}} \]

Variance component

Decrease \(SE(\widehat{ATE})\) with alternative research designs

Block-randomization

Repeated measures

Pre-treatment covariates

Pair-matched design

Online balancing

Sequential blocking

Rerandomization

Matching

Improving precision in experiments

\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{N-1}} \]

Variance component

Decrease \(SE(\widehat{ATE})\) with alternative research designs

Block-randomization

Repeated measures

Pre-treatment covariates

Pair-matched design

Online balancing

Sequential blocking

Rerandomization

Matching

All require pre-treatment information

Improving precision in experiments

\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{N-1}} \]

Variance component

Decrease \(SE(\widehat{ATE})\) with alternative research designs

Block-randomization

Repeated measures

Pre-treatment covariates

Pair-matched design

Online balancing

Sequential blocking

Rerandomization

Matching

All require pre-treatment information

Two categories:

  1. Reduce variance in observed outcomes

  2. Reduce variance in potential outcomes

Improving precision in experiments

\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{\color{#00843D}{N-1}}} \]

Sample size component

Improving precision in experiments

\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{\color{#00843D}{N-1}}} \]

Sample size component

Quadruple to halve \(SE(\widehat{ATE})\)

Focus: Increasing numerator may come at the cost of decreasing denominator

Precision gains from alternative designs may be offset by sample loss